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3. ABSTRACT 


Personal computer based flight simulators are expanding opportunities for 
providing low-cost pilot training. One advantage of these devices is the opportunity to 
incorporate instructional features into training scenarios that might not be cost effective 
with earlier systems. Research was conducted to evaluate the utility of different 
instructional features using a coordinated level turn as an aircraft maneuvering task. 

In study I, a comparison was made between automated computer grades of 
performance with certified flight instructors’ grades. Every one of the six student 
volunteers conducted a flight with level turns at two different bank angles. The 
automated computer grades were based on prescribed tolerances on bank angle, airspeed 
and altitude. Two certified flight instructors independently examined the video tapes of 
heads up and instrument displays of the flights and graded them. The comparison of 
automated grades with the instructors’ grades was based on correlations between them. 

In study II, a 2x2 between subjects factorial design was used to devise and 
conduct an experiment. Comparison was made between real time training and above real 
time training and between feedback and no feedback in training. The performance 
measure to monitor progress in training was based on deviations in bank angle and 
altitude. The performance measure was developed after completion of the experiment 
including the training and test flights. It was not envisaged before the experiment. The 
experiment did not include self- instructions as it was originally planned, although 
feedback by experimenter to the trainee was included in the study. 
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8. INTRODUCTION 


Advancements in computer technology have made the use of computer based 
simulators and trainers more feasible for research investigating factors related to pilot 
training effectiveness. Because the advantages of training on flight simulators includes 
savings in time and money, computer based training provides researchers with increased 
opportunity to investigate such factors that may affect flying and pilot training. Previous 
research by this team of researchers has investigated several techniques related to training 
pilots; they include: the use of different forms of automated feedback (see, for example, 
Ali, Guckenberger, Rossi and Williams, 2000) and the use of above real time training 
(ARTT) (see, for example, Rossi, Crane, Guckenberger, Ali, Archer and Williams, 1999) 

These studies have found that some values of ARTT and some forms of feedback 
are beneficial for training some piloting tasks. However, many questions regarding the 
use of ARTT, feedback and other variables remain to be studied. For instance, some 
questions are to determine precisely which values of ARTT are most effective for 
different tasks and whether ARTT is beneficial for all tasks. With respect to feedback, 
questions remain regarding whether automated feedback, (and of what nature) can be 
adequately provided to improve training of pilots. In addition, in order to maximize the 
potential usefulness of a simulator for training, the development of performance measures 
that adequately capture the pilot’s performance and assess training improvement is 
essential to the development of a simulator that can provide automated feedback to the 
trainee. This report, therefore, addresses these issues through two separate studies 
focusing on the teaching of a coordinated level turn. 

The first study was an attempt to validate automated performance measures to 
adequately capture flying performance and training improvement for novice pilots. The 
second study addressed the use of ARTT vs. Real Time Training (RTT) and the use of 
post-flight feedback vs. no feedback for training of a coordinated level turn. Each study 
will be discussed separately. The Literature Review that follows addresses the three areas 
of Above Real Time Training, Self-Instructions, and Development of Performance 
Measures separately. 

9. LITERATURE REVIEW 

9.1: ABOVE REAL TIME TRAINING 

Above Real Time Training is the training acquired on a real time simulator when 
it is modified to present events faster than normal. Crane and Guckenberger (2000) have 
provided a survey of the research work on ARTT, which covers its effects on training of 
novices and experienced individuals, and the types of algorithms needed to implement it 
on a simulator. Kolf (1973) noted that, "regardless of type or amount of pre-flight 
simulator training accomplished by the pilot, the actual flight appears to take place at 
much faster time frame than real time." Hoey (1976) reported that the mental state of test 
pilots operating remotely piloted vehicles can be approximately simulated without 
stressful conditions by increasing the simulated rate of time passage. Guckenberger et al 
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(1997) placed the objectives of NASA Dryden Flight Research Center and Air Force 
Fluman System Center's Technical Planning Integrated Product Team in perspective. For 
the training of pilots, they proposed several benefits from ARTT including increased rate 
of skill acquisition, decreased real time workload and decreased real time stress. Crane 
and Guckenberger (1997) have reported that pilots trained using ARTT performed 
emergency procedures and defeated bandit aircraft significantly faster than pilots trained 
in real time. 

Rossi, Crane, Guckenberger, Ali, Archer and Williams (1999) trained university 
students on a gunnery task to compare RTT and ARTT at 1 .5 times real time. The 
students trained in ARTT performed on test trials as well as students trained in RTT, 
although the ones trained in ARTT spent less clock time. During training the performance 
of students in ARTT was depressed compared to those trained in RT, however. The 
authors suggested that using ARTT as top-off training after RTT might result in more 
effective training. Williams (1999) also observed that ARTT as top- off training after 
RTT offers better training in comparison to ARTT alone or RTT alone in a similar 
gunnery task. 

In an Air Force sponsored study, Ali, Guckenberger, Rossi, and Williams (2000) 
addressed the use of ARTT for training of pilots to perform basic flight maneuvers. They 
classified the flying maneuvers with reference to Fitts and Posner's (1967) model that 
recognizes different stages of skill acquisition as cognitive, associative and autonomous. 
Ali et al (2000) observed that flying straight and leve 1 is relatively less complex than 
performing climbs and descents and level turns. For the training of straight and level 
flying they suggested using ARTT at the cognitive stage or the use of ARTT alone as a 
beneficial mode of training. For the training of climb and descent and level turns, 
however, they propose that ARTT is beneficial at the autonomous stage or the use of 
ARTT is recommended as top-off training after RTT. 

9.2: SELF-INSTRUCTIONS 


Self- instruction through the use of feedback is increasingly becoming beneficial 
for Computer Based Learning. Proctor and Dutta (1995) provided a comprehensive 
discussion of the influence of feedback on motor skill acquisition. Ali et al (2000) used 
in-flight and post- flight automated feedback as a form of self-instruction. In-flight 
feedback consisted of audio cues that signaled trending out of the prescribed tolerances in 
altitude and heading. Post- flight feedback consisted of showing the student a set of strip 
charts upon completion of a flight. These strip charts provided a graphical comparison of 
prescribed tolerances with the values of altitude, heading, airspeed, and other parameters 
reached in a student's flight. A progressive investigation of the effects of different forms 
of self-instruction for training of flying skills on a simulator is likely to bring significant 
improvements in the training of pilots. In general, participants who received feedback 
performed better than their counterparts who did not receive feedback. 
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9.3: PERFORMANCE MEASURES 


Straight and level flight, climb, descent and level turn are regarded as the four 
basic flight maneuvers in a Federal Aviation Administration (1995) publication on 
private pilot practical test standards. For evaluation of a trainee’s flight, it prescribes 
certain tolerances in the basic flight parameters. Following FAA’s guidelines, a certified 
flight instructor evaluates the performance of a trainee by directly observing his or her 
flight of an airplane. 

For modem training simulators, however, Vreuls and Obermayer (1985) 
emphasize the need of automated performance measures as a substitute for the evaluation 
of performance by direct observations. They present several benefits of the automated 
performance measures. Measurements can be based on a larger number of factors than is 
possible through direct observation, and precision and reliability can be improved. Data 
can be collected, summarized and analyzed in a short period of time. Adapted feedback 
and automated training can be implemented and personnel requirements can be reduced. 
The authors, however, recognize the need of research for devising automated 
performance measures, validating them and obtaining the listed benefits from them. For 
validating automated measures, they have suggested several ways, including the need of 
experts to judge performance quality, then determine which measures correlate with 
experts’ judgments. For automated grading, Vogel (2000) suggests awarding of grade 
points 4, 3, 2, 1, and 0 on flying performances within the given tolerances. 

Ali, Guckenberger, Rossi and Williams (2000) used grade points for flying 
performances on the simulator. For example, positive and negative tolerance for air speed 
in knots was 3 for 4 points, 6 for 3 points, 9 for 2 points and 12 for 1 point. For 
automated scoring on the computer, the computer monitored flight parameters at every 3 
seconds. To validate the automated scoring, several flights flown on a simulator were 
simultaneously evaluated by the computer and by a certified instructor. The comparison 
of the two kinds of evaluation revealed acceptable correlation for straight and level 
flights but very low correlation for climb, descent and turning maneuvers. In the referred 
report, the authors described the details of their method of automated scoring of a 
trainee’s flight. 

Ali, Khan, Rossi, Crane, Guckenberger and Bageon (2001) proposed that a 
performance measure which represents an increase or decrease of performance at 
different stages of training is a valid measure to assess progress in training even if the 
measure is not adopted for certifying a trained pilot. In their study, a trainee on the 
simulator was required to follow a prescribed ground track while maintaining required 
speed and altitude. At every prescribed instant of time in a flight, they used a single 
parameter for representing the performance; the parameter value was then averaged over 
the complete flight to obtain a consolidated performance measure. The single parameter 
was the magnitude of the displacement vector between the actual location and the 
prescribed location of the simulated flight at a given instant. 
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9.4: INFERENCE FROM LITERATURE REVIEW 


The literature review reveals that many questions regarding the utility of ARTT 
for various flying maneuvers are unanswered. Also, the use of different values of ARTT 
has not been systematically studied for novice pilots. Both of these issues have yet to be 
resolved in the context of simulator training. Further, the development of performance 
measures that adequately capture the pilot’s performance and assess training 
improvement is essential to the development of a simulator that can provide automated 
feedback to the trainee. Such automated performance measures provided in the form of 
feedback will maximize the potential usefulness of a simulator for training, 

Therefore, this report addresses the issues mentioned above in the context of the 
training of a coordinated level turn. The first study addresses the evaluation of a 
performance measure to be used as automated feedback and the second study addresses 
the effects of feedback vs. no feedback in conjunction with different values of ARTT. 
Same equipment was used for both studies. Therefore the equipment is described prior to 
the description of the two studies. 

10. EQUIPMENT 

Mock setup of a partial cockpit housed in the Flight Vehicle Lab at Tuskegee 
University was used as the pilot training station. On this setup, a student gets a panoramic 
view on three monitors for the out-the-window (OTW) display and a fourth monitor for 
instrumentation panel located below the OTW monitors. Among the three OTW 
monitors, the center monitor has the heads up display (HUD). The four monitors are 
governed by a Heavy Metal Computer acquired from Quantum 3D, Lake Forest, 
California. The computer was configured by SDS International, Orlando, Florida. It has 
two Pentium II 400 MHz processors, 400 MB RAM, three extra display cards for Open 
GVS, based graphics and a Sound Blaster audio card. The flight simulation software is 
the Lite Flite version 3.3 (ref. Lite Flite, 1999) available from SDS International. Lite 
Flite offers flight simulation of several aircraft including a Predator unmanned air vehicle 
(UAV). The Predator (UAV) simulation was used in these studies. The joystick was a 
Saitek X36F. It had labels taped on each of the surrounding four comers indicating the 
function of the corresponding movement; they were “Left Wing Up”, Right Wing Up”, 
“Nose Up”, “and “Nose Down”. The throttle was a Saitek X35T and had the two labels in 
the appropriate positions, “Increase Power”, “Decrease Power.” The rudder pedals were 
CH pedals. More details on equipment and its fidelity are given in Williams (2000). 
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Study I - A Comparison of Automated Performance Measures and Instructor 
Grading of Level Turns 

11.1 FLIGHT DATA FOR GRADING 

An important element of the work under the grant was to conduct a comparative study of 
the automated grading of the performance generated by the computer and the grading by 
certified flight instructors. For this purpose six flights by novice pilots were conducted 
consisting of a 1 80 degree level coordinated turn with a bank angle of 30 degree followed 
by another 1 80 degree turn in the opposite direction with a 45 degree bank angle. The 
heads-up display (HUD) and heads-down display (HDD) of these flights were videotaped 
for post- flight analyses by the instructors. 

11.2 AUTOMATED GRADING 

The automated grading criteria were based on the requirements of 90 knots ±3knots in 
airspeed, 30 or 45 degrees ± 3 degrees in bank angle and 10000 ft ± 50 feet in altitude. 
Flight parameters within these limits were graded as ‘A’. Deviations of ± 6 knot in 
airspeed, ± 6 degrees in bank angle and ± 100 feet in altitude were graded as ‘B\ 
Deviations of ±9 knots in airspeed, ±9 knots in bank angle and ±150 knot in altitude were 
graded as ‘C’. Deviations of ±12 knots in airspeed, ±12 degrees in bank angle and ±200 
feet in altitude were graded as ‘D’. And, deviations of < - 12 knot and > ±12knot in 
airspeed, < - 12 degree and > ±12 degree in bank angle and < -200 feet and > ±200 feet in 
altitude were graded as ‘F\ The flight parameters were graded every three seconds. An 
average grade was then calculated for each turn. 

11.3 INSTRUCTORS’ GRADING 

The instructors were asked to review the videotapes of the turns and evaluate the 
performance of the novice pilots on flight simulator. They used the criteria based on their 
own respective experiences of grading pilots for Federal Aviation Administration (FAA) 
certification and their own perception of evaluating novice trainees. They were asked to 
grade the flight parameters of bank angle, airspeed and altitude. The instructors then 
provided an average performance grade for each of the turns based on these individual 
grades. 

11.4 COMPARISON OF GRADES 

The comparisons of the overall automated vs. the instructors’ grades for each of the turns 
are shown as scatter charts in Figs. 1 . 1 and 1 .2. As can be seen from the scatter charts, 
most of the scores are in the 2.0 or below range. This was expected, as the subjects were 
novice pilots. However, the two instructors’ scores were somewhat inconsistent with one 
another. In general, instructor# 1 graded higher in comparison with the automated grades, 
while the grading of instructor#2 was lower than the automated grades. 
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Table 1.1: Correlation Matrix for Average Turning Performance 
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Figure 1.1: Scatter Plot of Instructor One Grade vs. Computer Automated Grade 
(Turn Task) 
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Figure 1.2. Scatter Hot of Flight Instructor Two Grade vs. Computer Automated 
Grade (Turn Task) 

The automated and instructor grades for the individual flight parameters (i.e. 
airspeed, altitude and bank angle) were also compared. The correlations of the automated 
vs. the Instructors’ grades were high, but the correlation between the instructors is 
moderate except for the altitude parameter as the pilots were consistently losing altitude. 
A higher correlation of instructor# 1 grading with the automated grade of bank angle is 
observed, whereas for the other two parameters instructor# 2 had a higher correlation 
with the automated grade. 


Table 1.2: Correlation Matrix for Grades on Airspeed: 
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Table 1.3: Correlation Matrix for Grades on Altitude: 
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Table 1.4: Correlation Matrix for Grades on Bank Angle: 
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Figure 1.3: Instructor vs. Automated Grading (Airspeed) 
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Figure 1.4: Instructor vs. Automated Grading (Bank Angle) 

11.5 COMMENTS 

This analysis points to the possibility that a comparison between automated and 
instructor grades may not be valid for novice pilots who have a wide variation in their 
performance. It also suggests that letter-grades may not be appropriate for tracking 
training progress as for novices the improvements are incremental. Thus they may exhibit 
improvements in controlling individual flight parameters but still the overall ‘letter’ grade 
may not show an improvement in overall skill. Although the correlations are moderate to 
high between instructors, the instructors both seem to have a systematic difference 
between their score and the computer score. Further, the differences in correlations on 
different parameters between instructors and the computer grade suggests that perhaps, 
the instructors are making evaluations based on different factors from each other. Thus 
the factors influencing the assessments of the instructors need to be understood. Future 
studies could systematically address these issues. 
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Study II - The Influence of Feedback and ARTT on the Training of a Coordinated 
Level Turn 

12.1 METHOD 

The purpose of this experiment was to investigate the effects of ARTT in 
comparison to Real Time Training (RTT), and the effects of one form of feedback vs. no 
feedback during the training of a standard two -minute turn with novice pilots. 

12.1.1: PARTICIPANTS 


Participants in the experiment were twenty undergraduate college students 
enrolled at Tuskegee University. The students were offered the incentive of extra course 
credit in either their psychology course or their engineering course in exchange for their 
participation in the experiment. Participants who had either no prior flying experience or 
few flying lessons were recruited. Actual pilots were not allowed to participate in the 
experiment. Those participants then were placed randomly in one of four groups selected 
by the experimenter, comprising five in each group. As described in the training section, 
four different groups were identified according to their respective training programs. 

12.1.2: EXPERIMENTAL DESIGN 


This experiment consisted of a 2 X 2 between- subjects factorial design. The first 
variable was the type of training, with the two levels being Real Time Training and 
Above Real Time Training. The second variable was Feedback vs. No Feedback. There 
were five participants in each group, with the exception of four in a 1. 0/1.0 group 
(Groups are described in the training section). The data from the fifth participant in the 
1 .0/1 .0 group was dropped as the score was an outlier. An examination of the student’s 
data sheet revealed that he had complained of a headache and that the experimenter noted 
his lack of interest in the study. 

12,1.3: PROCEDURE 

Participants were asked to reserve approximately two hours of their time for the 
completion of the experiment. Each participant completed the experiment alone. Upon 
arrival at the laboratory, each participant was asked to complete a consent form 
(Appendix A) indicating a desire to participate in the experiment. The participant was 
then given a Background Survey (Appendix B) to be completed to give insight on any 
experience with previous flight training. All participants then experienced the following 
segments of training and testing to be described in detail below: 

Orientation to Simulator Controls and Functions 

Demonstration Flight 

Familiarization Flights 

Training Flights 

Test Flights 

Debriefing 
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Orientation to Simulator Controls and Functions 

During the familiarization process, participants were informed of the basic parts 
of the aircraft, their functions, and the movements associated with each part. The 
participants then were given an overview of the locations and fimctions of the joystick, 
rudder pedals, and the throttle located in the mock cockpit. Then, participants were 
instructed in the location and functions of the following displays on the Heads Up 
Display (HUD): altimeter, radio altimeter, airspeed indicator, heading tape, artificial 
horizon, pitch ladder and clock. They were also instructed on the location and functions 
of the following instruments on the Heads Down Display (HDD): artificial horizon and 
the vertical velocity indicator. Because the turn and slip indicator on the HDD was not 
functioning properly, the use of HUD to determine if the aircraft was making a 
coordinated turn was explained. 

Demonstration Flight. 

The experimenter then demonstrated a typical flight and the participants were 
allowed to control the plane with the help of the experimenter. During this demonstration 
the experimenter covered essential controls of the aircraft and explained how to maintain 
airspeed heading and a certain bank angle. 

Familiarization Flights 

Participants then were told that they would engage in a series of familiarization 
flights. Participants flew four three-minute flights of three minute duration each in which 
they were told to maintain airspeed of 129 knot, a heading of 360 degrees, and an altitude 
of 5,000 ft. During all but the last flight, the experimenter coached the participant. For the 
first familiarization flight, the experimenter told the participant to focus primarily on 
maintaining altitude. Also, the experimenter assisted with the controls, if needed. During 
the second flight, the experimenter told the participant to focus primarily on maintaining 
altitude and heading, but the experimenter did not touch the controls. During the third 
flight the experimenter told the participant to focus on the altitude, heading and airspeed. 
Finally, on the fourth flight the participant was not coached, but told to maintain the 
airspeed, heading and altitude. 

During familiarization flights, participants were graded as if on a 4.0 grade point 
average (GPA) scale. These scores were computed by a visual basic code that compared 
parameter values in the flight with the parameter values and tolerances in an input file. 
For the 4.0 grade, the tolerances in airspeed, altitude and heading respectively were ±6 
knots, ±100 ft and ±2 degree. The grading procedure was same as described in section 
1 1.2 on Automated Grading. Students who achieved a score of 2.0 on at least one 
familiarization flight continued into training. Students who did not achieve the minimum 
score of 2.0 were told that the experiment was concluded. They were debriefed and 
thanked for their participation. Twenty six participants completed familiarization flights. 
Twenty participants, who achieved the passing grade continued on to the training phase. 

Training 

Training consisted of ten sessions, in each session a participant was required to 
perform a coordinated level turn of 180 degree while maintaining 10 degree bank angle 
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and 129 knots airspeed at 5,000 ft altitude. A participant received training under one of 
the following four different training programs: 

1.5/2.0 NoFB: A participant conducts the first five training flights at 1.5 ARTT 
(Simulation presents events 1.5 times faster than normal) and the next five training flights 
at 2.0 ARTT. No post flight feedback is provided. 

1.5/2.0 FB: A participant conducts the first five training flights at 1.5 ARTT and the next 
five training flights at 2.0 ARTT. The participant also receives post flight feedback. 
1.0/1.0 NoFB: A participant conducts all the ten training flights at RTT. No post flight 
feedback is provided. 

1.0/1.0 FB: A participant conducts all the ten training flights at RTT. The participant also 
receives post flight feedback. 

Participants in the 1.0/ 1.0 groups completed each session in three minutes. 
Participants in the 1. 5/2.0 groups completed a 1.5 ARTT session in two minutes and a 2.0 
ARTT session in one and a half minutes. Thus, all participants completed 10 sessions, but 
the participants in the ARTT groups completed training more quickly. 

Participants were given the following scenario to provide a context for the 
training sessions. 

Training Flights Briefins Scenario 

You’re a flight leader of a combat air patrol on a routine mission at 5000ft 
altitude, 129 knots and heading in a 360 direction. An unidentified low speed aircraft is 
being tracked by radar moving towards a no fly zone. Despite repeated requests, the 
aircraft fails to identify itself and continues towards the. no fly zone. The Air Defense 
Radar Controller assigns the interception to your flight in order to make a visual 
identification. You are asked to initiate a right hand turn, maintain a bank angle of 10 
degrees, an altitude of 5,000 ft and a speed of 129knot. Exactly after two minutes (as 
recorded on your heads up display) you should level out. Your heading should be 180 
degree, putting you right behind the target close enough for visual contact. You will 
complete a number of these missions. Do your best, because once you have finished the 
practice missions, you will be required to complete a test mission with a slightly more 
difficult task. 

Feedback in Training 

Participants in the NoFB groups received no feedback during or after flights. If 
they asked questions about the task, they were reread the relevant instructions. 
Participants in the FB groups received verbal feedback after each flight. In verbal 
feedback, the experimenter told the participant the errors in manipulating the controls and 
in focusing on the wrong instruments. After the second, fourth, sixth, eighth, and tenth 
flights, participants in the FB groups were also shown a screen printout of the desired 
ground track and the pilot’s actual ground track for comparison. Then, the instructor 
explained possible reasons for the deviations in performance. 
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Testing 


Upon completion of training, participants were told that they would now complete 
two flights with a slightly different and more challenging task. These two flights were 
conducted in real time for all participants and each lasted for three minutes. No feedback 
or instruction was provided during testing. Participants were read the following scenario 
prior to each flight. 

Testing Scenario 

As part of a Combat Air Patrol, the Air Defense Radar Controller notifies you that 
a hostile aircraft is being tracked headings towards a vulnerable point (VP). Your 
instructions are to initiate a right hand turn in order to pursue the aircraft while 
maintaining a bank angle of 30 degrees, an altitude of 5,000ft., and a speed of 164 kts. 
You then are informed of a second target. Interception of the second target is also 
assigned to your flight as a priority two target. Exactly 48 sec. after instructions to initiate 
the turn you should level out. You will now be exactly behind the target on a heading of 
180. You should fire your missile and immediately initiate a left tim with a 30-degree 
bank maintaining 5,000 ft. altitude and a speed of 164kts. In pursuit of the second target 
assigned. Exactly 48 sec after initiating the second turn you should level out. Your 
heading should be 360, putting you right behind the second target. Launch your missile. 
Mission accomplished. 

Upon completion of the two flights, the participants were thanked and engaged in 
a debriefing session with the experimenter. 

12.2 PERFORMANCE INDEX (PI) 

In the present study, a trainee conducted several flights of the same kind. A flight 
required completion of a 180-degree level turn while maintaining the required bank 
angle, airspeed and altitude. It is understood that the required level turn follows a semi- 
circular flight path. At the outset, it was envisaged that the performance measure used by 
Ali et al (2001) was an appropriate measure for this study. Accordingly, the magnitude of 
the displacement vector between the actual location and the prescribed location of the 
simulated airplane at a given instant, in other words the deviation from the desired flight 
path, was expected to be an appropriate performance measure. But most of the flights 
conducted by the trainees showed little correspondence with the semi-circular flight path. 
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Figure 2.1. Turn Study. 180-degree-Turn paths (RT) in a training session, 
turns 00404. 


Figure 2.1 shows the desired flight path in a 180-degree turn and the actual flight 
path of a typical flight performed by a participant in a training session. Fig 2.2 shows the 
desired flight path in an S turn and the flight path of a typical flight performed by a 
participant in a test session. 

Observations of the actual flights led the investigators to realize that the out of the 
window (OTW) display, HUD and HDD did not provide sufficient cues for the trainee to 
gain situation awareness, thereby making the execution of the 180-degree turn during 
training rather difficult and execution of the S turn during testing nearly impossible (Fig. 
2.2). Therefore, the deviation from the desired flight path was not considered to be a valid 
performance measure. 
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Figure 2.2.Turn Study. S-Tum paths in a test session (RT) turns_00404 

However, irrespective of the flight path, the trainees attempted to maintain the 
required bank angle, speed and altitude. Therefore, a deviation based on the root mean 
square (rms) value of the dimensionless representations of the differences in the actual 
and desired values of bank angle, air speed and altitude was considered to be a more 
realistic measure of the performance of the subjects. At any given instant in a flight, this 
performance index is defined by: 

PI = v [(A<()) 2 + (Ah/R) 2 + (AV/R) 2 ], where 

A<(> = deviation in the bank angle from the desired bank angle over the turn duration 
Ah = deviation in altitude from the desired altitude over the turn duration 
AV = deviation in velocity from the desired speed over the turn duration 
R = theoretical radius of turn for the desired speed and bank angle given by the relation 
R= V 2 /(gtanj>), where V= desired speed and g is the acceleration due to gravity. 

In general, a deviation in speed must be included in the calculation of the PI. 
However for this specific study, it was observed that a deviation in velocity and altitude 
were directly correlated (Fig. 2.3 and 2.4). This primarily was due to the fact that the 
pilots were not manipulating the throttle during the turn. Hence, inclusion of only the 
deviation in altitude also captured the effect of poor speed control. 
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Figure 2.4.Turn Study. Altitude Deviation with Time (turns_00295) 

We suggest that this performance index may be used as an automated 
performance measure for future studies. We did not use it as an automated measure in 
this study, however, because the index was developed after completion of the study 
during data analysis. 
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The PI developed for this study, as described above, is used in the analyses 
reported below. It is understood that a decrease in the deviation implies improvement in 
performance. For a flight, the PI is the time average ofthe deviations at different instants 
in the flight. For training flights of a participant, the averaging of PI was done for the 
complete flight. For testing flights of a participant, the averaging of PI was done for the 
first 1 80-degree turn of the flight instead of covering the full S turn. It implies that the 
requirement of the second half of the S turn was dropped from the evaluation due to lack 
of cues as pointed out earlier in this section. 

12.3 RESULTS 

As mentioned in the training and test sections, a participant completed ten training 
flights and two test flights. As explained in the Performance Index (PI) section, the PI for 
a participant’s flight represents an average of a combined measure of bank angle and 
altitude deviations in the flight. Thus a decrease in the PI represents improvement in 
performance. Fig 2.5 provides PI values for training flights numbered 1 to 10 and for test 
flights numbered 1 and 2. Every point graphed in Fig 2.5 represents PI value for the 
respective flight averaged over all the participants in the respective group. For example, 
the average value of PI for the five participants in the 1. 5/2.0 NoFB group in their third 
training flight is 0.2. 

A three-way , split-plot Analysis of Variance (ANOVA) was conducted with one 
between-subjects factor, test trial 1 vs. test trial 2, and two within-subjects factors, 
training time and feedback condition. There was no significant difference in performance 
scores between the two test trials, F(l, 15) <1 , and interactions between test trials and 
training time, feedback condition, and test time by feedback condition interaction were all 
not significant (all F values < 1). For the between-subjects factors there was no 
significant interaction between training time and feedback condition, F (1, 15) = 2.99. 
However, there was a significant difference in performance scores between feedback and 
no feedback groups, F (1, 15) = 6.418, p < .05. Specifically, participants in the feedback 
groups performed better than those in the no feedback groups on test trials as shown in 
Fig. 2.6. In addition, there was a significant difference in performance scores for training 
time, i.e., participants in the RT/RT groups performed better than those in the ARTT 
groups on test trials, F(1 , 1 5) = 7.009, p < .05 as shown in Fig 2.7. 

12.4 DISCUSSION 

The various training interventions also were observed to influence performance 
across training flights (Fig. 2.5) in addition to the test trials discussed above. In general, 
participants in both of the feedback groups improved performance early in training, and 
performed better than their counterparts across training sessions. This finding is 
consistent with the ANOVA of test trial performance. The group who received ARTT 
without feedback (1. 5/2.0 NoFB) did not exhibit improvement with time. It is possible 
that the improvement that may be experienced through practice alone was masked by the 
increasing difficulty of changing from 1.5 ARTT to 2.0 ARTT. Evidence of this 
possibility comes from examining the performance of participants in the RTT without 


23 


feedback group (1. 0/1.0 NoFB). The participants who experienced RTT without feedback 
may be viewed as a control group and it may be seen that their performance improved 
with practice in the absence of extrinsic feedback; however, this improvement did not 
appear substantially until the eighth training session. Similarly, one can observe an 
improvement after initial sessions for the participants in ARTT/FB, but no substantial 
improvement in later sessions during the 2.0 segment of training. Thus, it seems that 
during RTT, feedback is not as necessary as during ARTT, although the use of feedback 
can greatly enhance performance early in training. Also, the use of ARTT may not be 
suitable for novice pilots early in their training. 



Figure 2.5 Average Performance Index for Each Group for Training and Test 
Flights. (Decrease in performance index implies improvement in performance). 
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Figure 2.6 Average test performances of Feedback and No Feedback groups. 
(Decrease in performance index implies improvement in performance.) 
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1.5- 2.0 1.0 -1.0 


ARTT Condition 

Figure 2.7 Average test performances of ARTT and RTT groups. (Decrease in 

performance index implies improvement in performance.) 
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13. CONCLUSION 


Two studies were conducted in the context of training of a coordinated level turn 
on a flight simulator. 

In study I, a comparison was made between automated grades of performance 
with certified flight instructors’ grades. The study pointed to the possibility that a 
comparison between automated and instructor grades may not be valid for novice pilots 
who have a wide variation in their performance. The study also suggested that novices 
may exhibit improvements in controlling individual parameters; therefore an overall 
grade may not capture the incremental improvements. 

In study II, the experiment consisted of a 2x2 between subjects factorial design. 
The first variable was the type of training with two levels being real time training (RTT) 
and above real time training (ARTT). The second variable was feedback versus no 
feedback. For training of the level turn, a performance index based on deviations in the 
bank angle and altitude offered a valid measure to monitor improvement in performance. 
On the test trials, the participants in the feedback groups performed better than those in 
the no feedback groups; and participants in the RTT groups performed better than those 
in the ARTT groups, but the training was accomplished in less clock time for the ARTT 
groups. If ARTT is to be used for training novices to perform level turns on a simulator, 
then it should be used in conjunction with appropriate feedback and it should be used 
after the trainee has attained adequate improvement in performance to have reached a 
plateau. 

14. RECOMMENDATIONS FOR FUTURE WORK 

The present two studies addressed the training and measurement of a coordinated 
level turn using a PC-based flight simulator. Several issues from each study have been 
raised and may be addressed through future studies. 

First, the development of an adequate performance measure to capture flying 
performance and the improvement of flying performance remains a challenge. In Study I, 
the automated performance measures involving deviations from prescribed tolerances 
resulting in GPAs were correlated fairly well with the instructor’s grading of such turns. 
However, the differences between the instructor’s grades and the computer grade may be 
minimized in future studies. In addition, the fact that the two instructors’ grades were 
systematically different from one another and from the computer grade suggests that the 
measure may not adequately capture the pilot’s performance. Also, future studies could 
determine what factors are being used by the instructors when grading. For example, our 
analysis suggests that instructors may be focusing on different parameters of the flight. 

In study II, the researchers developed a new index to evaluate flying performance 
that appears to track training progress well. Future work using this new performance 
index could include: 


26 



(i) Evaluation of this index in comparison to certified flight instructors and other 
performance measures, (ii) using this performance index as an automated feedback 
measure in the training of various piloting tasks, and (iii) using this performance index 
and comparing novice pilots to experienced pilots when performing turns and other 
maneuvers. 

The study II also left unanswered several questions. Since participants in the 
ARTT groups did not perform better than those in the RTT groups there are several 
possible explanations: 

First, the use of ARTT for this maneuver is not effective.Second, the change from 1.5 
ARTT to 2.0 ARTT masked improvement in performance. 

One might suggest, therefore, that if ARTT is to be used, it should be used as top- 
off training, i.e., after the pilot has experienced sufficient improvement in performance to 
have reached a plateau. If ARTT is to be used, that it be used in conjunction with 
appropriate feedback. 
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16.1 INFORMED CONSENT FORM 


DEVELOPMENT AND ASSESSMENT OF A NOVEL TRAINING PACKAGE USING 
SELF INSTRUCTION METHODS AND ABOVE REAL TIME TRAINING (ARTT) 
FOR BASIC MANEUVERING TASKS ON A FLIGHT SIMULATOR 

As an undergraduate student of Tuskegee University, you are invited to participate as a 
trainee in an experimental research study. The study will develop and evaluate a new 
program of training on a flight simulator. 

Your participation will require approximately two hours of flying on a simulator one day. 
You will take a seat as a pilot in a mockup cockpit located in the flight vehicle design 
laboratory on Tuskegee’s campus. You will control the flight movements of the joystick, 
throttle and rudder pedals while watching the flight parameter changes on the simulated 
instruments panel Heads Up Display (HUD) and the moving icon of the airplane on the 
computer screens. The available instructor will give you a set of instructions, allow you 
some time for familiarization and for asking questions, and provide feedback on your 
performance whenever necessary. 

No physical or medical testing risks are involved in the experiment. The experiment, 
however, is not official pilot training. As a possible risk, some of the simulator flight 
techniques that you will leam may not be applicable in real flying. Moreover, as it 
happens in any learning exercise, you may need to overcome temporary disappointments 
on possible lack of progress in performance, especially when your flight will face a 
simulated crash. 

In general, the training will be an enjoyment and a learning experience. The data 
provided by you on a survey form and the data on your performance in the training will 
be coded to protect your confidentiality. Your participation is voluntary and you may 
withdraw from the experiment at any time. Your participation is voluntary and you may 
withdraw from the experiment at any time. Your participation, according to your agreed 
upon schedule, however will be greatly appreciated by investigators. 

The successful completion of this research program will be a valuable contribution by 
Tuskegee University in improving the methods of training pilots on a simulator. 

Fro any questions or concerns, please feel free to contact the investigators on campus: Dr. 
Syed Ali at 727-8853, Dr. Marcia Rossi at 727-8830, or Dr. Muhammad Khan at 727- 
8637. In the event of an unresolved grievance, you may also contact Dr. Stephen Sodeke, 
the Chairperson of the Human Subjects Review Committee (HSRC) at 727-8363. 

The available investigator will give you a copy of this form to keep. By signing below, 
you are agreeing to participate in the study. 


Signature of Participant/ Date 


Signature of Investigator/ Date 
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16.2 Background Survey - Turn Study 

Code 

Major 

Sex: Male 

Female 


Age: 

Are you: Left Handed 

Right Handed 

Do you have any flight experience? 

Yes 

No 


If so, how many hours (estimate)? 

How long ago was your most recent flight training, if any? 

How many hours a week do you engage in video/ computer games? 

Light: 0-5 hours 

Medium: 6-13 hours 

Heavy: 13 or more hours 

What type of video / computer games do you play? 

Sports 

Fighting 

War 

Cards 

Flight 

Mystery 

Other 


What type of controllers do you use? 

Joystick 

Control Pad 

Keyboard 

Arcade 
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